Towards Accurate Markerless Human Shape and Pose Estimation over Time

We address the problem of accurately estimating human shape, pose, and motion from images and video without markers or special cameras. Existing methods often assume known backgrounds, static cameras, and sequence specific motion priors. Here we propose a method that is fully automatic and, given multi-view video, estimates 3D human motion and body shape. Our work is built upon the recent SMPLify [10] method, which fits a 3D body model to 2D joints detected by a CNN in a single image. We extend this in several ways. First we fit the body to 2D features detected in multi-view images. Second, we use a CNN method to automatically segment the person in each image and fit the 3D human shape and pose to the contours; this further improves accuracy. The 2D pose CNN sometimes confuses left and right sides of the body, causing large errors. Third we address this with a robust temporal prior term. Rather than use a learned, sequence specific prior, we use a generic DCT basis applied to the reconstructed 3D joints. We formulate this as a coherent objective function, giving a fully automatic markerless mocap system we call MuVS for Multi-View SMPLify. We compare with ground truth 3D shape and pose on the HumanEva and Human3.6M datasets. Our results are significantly more accurate than the state of the art and provide a realistic 3D shape avatar.